Generating reference files
In the Advanced tab of the Amazon S3 target endpoint, you can enable the Generate a reference file option. The Reference File contains a list of the Change File locations and is therefore only relevant if the task's Apply Changes or Store Changes options are enabled.
The format of the reference file name is as follows:
<amazon_s3_target_endpoint_name><counter>.csv|json
Example:
AmazonS300000001.csv
The counter suffix increases incrementally each time a new Reference File is generated (i.e. when the file reaches the maximum size defined in the General tab). Once a new Reference File has been generated, you can delete the old reference file(s) if required.
Whenever an Apply Changes data file is created, a new row is added to the Reference File in the following format:
<Source_Table_Name>,<bucket_name>/<path>/<file_name>
Example:
employees,bigdata/new/files/my.company/20170611-120144192.csv
Note that if the Post-process files option in the Advanced tab is also enabled, the Reference File will be generated after the post-processing completes.